Skip Headers

Oracle® Data Guard Broker 10g Release 2 (10.2) Part Number B14230-02 |
Home | Book List | Contents | Index | Master Index | Feedback

Previous | Next
| View PDF

# 10 Tro ubleshooting Data Guard

This chapter describes various errors and how to solve them. This chapter contains the following topics:

- Section 10.1, "Sources of Diagnostic Information"

- Section 10.2, "General Problems and Solutions"

- Section 10.3, "Troubleshooting Problems During a Switchover Operation"

- Section 10.4, "Troubleshooting Problems During a Failover Operation"

- Section 10.5, "Troubleshooting Problems with the Observer"

## 10.1 Sources of Diagnostic Information

The Data Guard broker provides information about its activities in several forms:

- Database status information (see Section 4.7)

- Oracle alert log files

  The broker records key information in the alert log file for each database. You can check the alert log files for such information when troubleshooting Data Guard.

- Data Guard "broker log" files

  For each database in a broker configuration, the broker DMON process records important behavior and status information in a "broker log" file, useful in diagnosing Data Guard failures.

  For example, the broker log files for all instances in a RAC are named as follows in a UNIX environment:

  `$ORACLE_HOME/rdbms/log/drc*.log`

## 10.2 General Problems and Solutions

This section describes general problems and solutions. This section contains the following topics:

- ORA-16596: Object Not Part of the Data Guard Broker Configuration

- Redo Accumulating on the Primary Is Not Sent to Some Standby Databases

- Many Log Files Are Received on a Standby Database But Not Applied

- The Primary Database is Flashed Back

### 10.2.1 ORA-16596: Object Not Part of the Data Guard Broker Configuration

A request was issued to the broker, but the database instance through which you have connected is no longer a part of the broker configuration. You may see this error when the broker monitor (DMON) fails to locate a broker configuration profile for the database upon which it is running.

Solution Reconnect to the configuration through another database instance. Confirm that the value of the `DB_UNIQUE_NAME` initialization parameter for the database and the database object name in the broker configuration are identical.

This problem can also occur if you attempt to enable a configuration, but the broker configuration file for one of its databases was accidentally removed or is outdated. In this case, remove the database from the broker configuration, manually delete the configuration file for that standby database (not for the primary database), and try again to enable the configuration. After the configuration is enabled, run the Add Standby Database wizard and choose the Add existing standby database option to create a new database profile for the previously deleted standby database.

### 10.2.2 Redo Accumulating on the Primary Is Not Sent to Some Standby Databases

By viewing the Log File Details page in Enterprise Manager, you have determined that log files are accumulating on the primary database and are not being archived to some of the standby databases in the broker configuration.

Solution To narrow down the problem, do the following:

- Verify that the state of the primary database is `ONLINE` (not `LOG-TRANSPORT-OFF`).

- Verify that the value of the `LogShipping` property of the standby database in question is `ON`.

- Check the status of the redo transport services on the primary database using the LogXptStatus monitorable property. If redo transport services have an error, then use the error message to determine further checking and resolution action. For example:

  - If the error indicates the standby database is not available, you need to restart the standby database.

  - If the error indicates no listener, you need to restart the listener.

  - If the error indicates the standby database has no local destination, you need to set up a standby location to store archived redo log files from the primary database.

### 10.2.3 Many Log Files Are Received on a Standby Database But Not Applied

By viewing the Performance page or Log File Details page in Enterprise Manager, you have determined that the standby database accumulates too many log files without applying them.

Solution There are many possible reasons why archived redo log files might not be applied to the standby database. Investigate why the log files are building up and rule out the valid reasons.

If the current status of the standby database is not normal:

- Determine whether or not the log apply services might be unexpectedly offline. See the ORA-16766 (for physical standby databases) or ORA-16768 (for logical standby databases) problem and solution statement for more help.

- If this is a logical standby database, check to see if a failed transaction has occurred.

- If you want to suppress the error while you investigate the problem, you can temporarily disable broker management of the database.

  See Also: Chapter 8 for additional information about disabling the database using the DGMGRL command-line interface

If the current status of the standby database is normal:

- Verify the state of the standby database is online (not in the `LOG-APPLY-OFF`, `OFFLINE`, or `READ-ONLY` states).

- Verify the state of the primary database is online (not in the `LOG-TRANSPORT-OFF` state).

- See Also: Chapter 9 for additional information about the `LogShipping` property

- Check to see if log files are building up because the value of the DelayMins property is set too large. (Log apply services will delay applying the archived redo log files on the standby database for the number of minutes specified.)

  See Also: Chapter 9 for additional information about the `DelayMins` property

- If you cannot see any errors, compare the archive rate to the apply rate on the Performance page in Enterprise Manager to see if the apply rate is lower than the archive rate.

### 10.2.4 The Request Timed Out or Enterprise Manager Performance Is Sluggish

If the broker requests are not completing within the normal timeout parameters, try the following actions to solve the problem:

1. Verify the network is operating appropriately.

2. Try to ping all of the nodes in the configuration.

3. Try reconnecting through another database to retry the operation.

4. Run the `VERIFY` command from the configuration to see which broker is busy. The output from the `VERIFY` command should show why (or at least which) broker is not able to process the requests.

### 10.2.5 The Primary Database is Flashed Back

If the primary database is flashed back, the standby databases in the configuration must be also be flashed back or reenabled to be viable targets for switchovers or failovers. The broker will report errors for the standby databases if the primary database has been flashed back.

For more information about restoring the viability of a standby database that was disabled by the broker, see Section 5.4.3.

## 10.3 Troubleshooting Problems During a Switchover Operation

If the switchover fails due to problems with the configuration, the broker reports any problems it encounters in the alert log files or in the broker log files (see Section 10.1). In general, you can choose another database for the switchover or restore the configuration to its pre-switchover state and then retry the switchover. The following subsections describe how to recover from the most common problems.

Problems Transitioning the Primary Database to the Standby Role

If the error status indicates a problem when transitioning the original primary database to the standby role (including stopping redo transport services and starting log apply services), use these general guidelines to recover:

1. Disable the configuration using DGMGRL.

   Note: You can enable or disable the configuration using DGMGRL. You cannot disable the configuration using Enterprise Manager. You can enable the configuration using Enterprise Manager if it was previously disabled using DGMGRL.

2. Investigate the error message returned by the broker to find the source of the problem on the primary database and correct it. Oracle Enterprise Manager provides an Alert Log Content link for viewing alert log file information. You may also examine the contents of the broker log file for the primary database.

3. Reenable the configuration to refresh and restore the databases to their original roles and states.

4. Perform the switchover again.

Problems Transitioning the Standby Database to the Primary Role

If the error status indicates that a problem occurred when transitioning the target standby database to the primary role (including stopping log apply services and starting redo transport services), use these general guidelines to restore to the pre-switchover state.

If fast-start failover is enabled, the broker does not allow failover to any standby database except to the target standby database. In addition, failover to the target standby database is allowed only under certain conditions:

- The observer is running

- The value of the `FS_FAILOVER_STATUS` column in the `V$DATABASE` view on the target standby database is either `READY` or `SUSPENDED`.

  See Also: Section 5.5.2.1, "What Happens When Fast-Start Failover and the Observer Are Running?"

Step 1 Analyze and correct the detected failure.

1. Disable the configuration using DGMGRL.

Note: You can enable or disable the configuration using DGMGRL. You cannot disable the configuration using the Enterprise Manager. You can enable the configuration using Enterprise Manager if it was previously disabled using DGMGRL.

2. Investigate the error messages returned by the broker to find the source of the problem on the standby database and correct it. Examine the alert log file information and the contents of the broker log file for the target standby database.

Step 2 Convert back to a primary database.

The original primary database was already converted to a standby database by the time this stage of the switchover is reached. The database must be converted back to being a primary database as follows:

1. Ensure the original primary database is at least mounted. A restart may be necessary:

       SQL> SHUTDOWN IMMEDIATE;
       SQL> STARTUP MOUNT

2. Execute the following SQL*Plus command to convert the database back to running in the primary database role:

   `SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;`

3. Restart the database if necessary. The database is now in its former role as a primary database.

Additional steps may be needed to allow any logical standby databases in the configuration to accept and apply redo logs from this primary database. See Oracle Data Guard Concepts and Administration for these steps.

Step 3 Finish the recovery.

1. Reenable the configuration.

2. Perform the switchover operation again.

Additional Problems that May Occur During a Switchover Operation

Problem: The switchover fails due to problems with redo transport services.

Solution:

1. Verify the state and status of the standby database by viewing its information on the Data Guard Overview page.

   See Also: Chapter 6 for additional information about the Data Guard Overview page

2. Run the Verify operation after the switchover completes to examine the alert log file of the new primary database and to verify the status of the configuration.

   See Also: Section 6.8.1 for additional information about verifying the configuration

Problem: The switchover may fail during verification checks done by Data Guard broker (for example, redo transport services might return errors on a database that is involved in the switchover).

Solution: Choose another database for the switchover or fix the problem by transporting the archived redo log files.

## 10.4 Troubleshooting Problems During a Failover Operation

A lthough it is possible for a failover to stop, it is unlikely. If an error occurs, it is likely to happen when the standby database is transitioning to the primary role. If the error status indicates that this problem occurred, use these general guidelines to fix the problem:

1. Investigate the error information provided by the broker to find the source of the problem and correct it.

2. Perform the failover again.

The following problems may occur during a failover operation:

Problem: The primary database is still running. Enterprise Manager tries to detect if the primary database is still running. However, if the primary database is still running and the Enterprise Manager cannot detect that it is running, the failover will not be successful.

Solution: If the primary database is still running but can no longer function as a primary database, shut it down and retry the failover.

## 10.5 Troubleshooting Problems with the Observer

The observer continuously monitors the fast-start failover environment to ensure the primary database is available. Installing and starting the observer is an integral part of enabling fast-start failover. The following sections describe techniques for troubleshooting the observer:

- Problems Starting the Observer

- Problems Because the Observer Has Stopped

- Capturing Observer Actions in the Observer Log File

### 10.5.1 Problems Starting the Observer

Only one observer can be associated with the broker configuration at any given time. If you attempt to start a second observer, one of the following errors is returned:

    ORA-16647: could not start more than one observer
    DGM-16954: Unable to open and lock the Observer configuration file

Use the DGMGRL `SHOW CONFIGURATION VERBOSE` command to determine the location of the observer that is currently associated with the broker configuration.

    DGMGRL> SHOW CONFIGURATION VERBOSE;
    Configuration
      Name:                DRSolution
      Enabled:             YES
      Protection Mode:     MaxAvailability
      Fast-Start Failover: ENABLED
      Databases:
        North_Sales - Primary database
        DR_Sales    - Physical standby database
                    - Fast-Start Failover target

    Fast-Start Failover
      Threshold: 45 seconds
      Observer:  observer.foo.com

    Current status for "DRSolution":
    SUCCESS

### 10.5.2 Problems Because the Observer Has Stopped

If the observer host machine crashes, the broker configuration is no longer observed and fast-start failover is no longer possible. In this case, you may have to move the observer to a new host if the original host machine cannot be repaired in a timely fashion. To move the observer, you must stop the first observer and then start a new observer on another host.

1. Issue the DGMGRL `STOP OBSERVER` command to sever the link between the original observer and the broker configuration:

       DGMGRL> STOP OBSERVER;
       Done.

2. Issue the DGMGRL `SHOW CONFIGURATION VERBOSE` and the `SHOW DATABASE` command to verify that the configuration is no longer being observed:

       DGMGRL> SHOW CONFIGURATION VERBOSE;

       Configuration
         Name:                DRSolution
         Enabled:             YES
         Protection Mode:     MaxAvailability
         Fast-Start Failover: ENABLED
         Databases:
           North_Sales - Primary database
           DRSales     - Physical standby database
                       - Fast-Start Failover target

       Fast-Start Failover
         Threshold: 45 seconds
         Observer:  (none)

       Current status for "The SUPER cluster":
       Warning: ORA-16608: one or more databases have warnings

       DGMGRL> SHOW DATABASE 'North_Sales';
       Database
         Name:            North_Sales
         Role:            PRIMARY
         Enabled:         YES
         Intended State:  ONLINE
         Instance(s):
           sales1
       Current status for "North_Sales":
       Error: ORA-16658: unobserved Fast-Start Failover configuration

3. Note that you do not need to issue the DGMGRL `SHOW` commands to verify that the observer has actually stopped. Successful completion of the DGMGRL `STOP OBSERVER` command will allow a new observer to become associated with the configuration.

   Now, you may start the new observer. After the observer is started, it prints its failover and reinstatement actions to the console in which it was started. For example:

       DGMGRL> START OBSERVER;
       Observer started

       14:41:23.95  Thursday, May 26, 2005
       Initiating fast-start failover to database "DRSales"...
       Performing failover NOW, please wait...
       Failover succeeded, new primary is "DRSales"
       14:41:38.62  Thursday, May 26, 2005

       14:42:15.43  Thursday, May 26, 2005
       Initiating reinstatement for database "North_Sales"...
       Reinstating database "North_Sales", please wait...
       Operation requires shutdown of instance "sales1" on database "North_Sales"
       Shutting down instance "sales1"...
       ORA-01109: database not open

       Database dismounted.
       ORACLE instance shut down.
       Operation requires startup of instance "sales1" on database "North_Sales"
       Starting instance "sales1"...
       ORACLE instance started.
       Database mounted.
       Continuing to reinstate database "North_Sales" ...
       Reinstatement of database "North_Sales" succeeded
       14:43:23.71  Thursday, May 26, 2005

### 10.5.3 Capturing Observer Actions in the Observer Log File

You can use the DGMGRL `-logfile` option to start the observer, so that all of the troubleshooting actions performed in Section 10.5.1 can be captured in a file. For example:

    % dgmgrl -logfile observer.log / "start observer"

All the observer output is then recorded in a file named `observer.log` in the current working directory where you issued the DGMGRL command.